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UPDATING ENTRIES 
CACHED BY A NETWORK PROCESSOR 

BACKGROUND 

[0001] A network communication system transmits information in packets from a 
5 transmitter to a receiver through one or more routers which route the packets 
between nodes within a network or between networks. The router may comprise 
one or more network processors to process and forward the packets to different 
destinations, and one or more external memories to store entries used by the 
network processors, such as node configuration data, packet queue and flow 
10 configuration data, etc. 

[0002] The network processor may comprise a control plane to setup, configure 
and update the entries in the external memories, and a data plane having a 
plurality of microengines to process and forward the packets by utilizing the 
entries. Each of the microengines may have a local memory to store entries of the 
15 external memories that are frequently used. Once the control plane updates 
entries in the external memory, it may send a signal to the microengine(s) of the 
data plane that may cache or store the updated entries In Its local memory. In 
response to the signal, the microenglne(s) may flush all entries stored in the local 
memory to make them consistent with entries stored in the external memory. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

[0003] The invention described herein is illustrated by way of example and not by 
way of limitation in the accompanying figures. For simplicity and clarity of 
illustration, elements illustrated in the figures are not necessarily drawn to scale. 
For example, the dimensions of some elements may be exaggerated relative to 
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other elements for clarity. Further, where considered appropriate, reference labels 
have been repeated among the figures to indicate corresponding or analogous 
elements. 

[0004] Fig. 1 shows an embodiment of a network device. 
(0305] Fig. 2 shows an embodiment of a network processor of the network device 
of Fig. 1. 

{0006] Fig. 3 shows an embodiment of a method implemented by a control plane 

of the network processor depicted In FIG. 2. 
(0007] Fig. 4 shows an embodiment of another method implemented by a 
10 microengine of the network processor depicted in FIG. 2. 
(0008] Fig. 5 shows a data flow diagram of an embodiment for updating entries 

cached by the network processor depicted in FIG. 2. 

DETAILED DESCRIPTION 

(0009] The following description describes techniques for updating entries cached 
15 in a network processor. In the following description, numerous specific details 
such as logic implementations, pseudo-code, means to specify operands, 
resource partitioning/sharing/duplication implementations, types and 
interrelationships of system components, and logic partitioning/integration choices 
are set forth in order to provide a more thorough understanding of the current 
20 invention. However, the invention may be practiced without such specific details. 
In other instances, control structures, gate level Circuits and full software 
instruction sequences have not been shown In detail in order not to obscure the 
Invention. Those of ordinary skill in the art, with the included descriptions, will be 
able to implement appropriate functionality without undue experimentation. 
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[001 0] References in the specification to "one embodiment", "an embodiment", "an " 
example embodiment", etc., indicate that the embodiment described may include 
a particular feature, structure, or characteristic, but every embodiment may not 
necessarily include the particular feature, structure, or characteristic. Moreover. 
5 such phrases are not necessarily referring to the same embodiment. Further, 
when a particular feature, structure, or characteristic is described in connection 
with an embodiment, it is submitted that it is within the knowledge of one skilled In 
the art to effect such feature, structure, or characteristic in connection with other 
embodiments whether or not explicitly described. 
[001 1] Embodiments of the invention may be implemented in hardware, firmware, 
software, or any combination thereof. Embodiments of the invention may also be 
implemented as Instructions stored on a machine-readable medium, that may be 
read and executed by one or more processors. A machine-readable medium may 
include any mechanism for storing or transmitting information in a form readable 
15 by a machine (e .g., a computing device). For example, a machine-readable 

medium may include read only memory (ROM); random access memory (RAM); 
magnetic disk storage media; optical storage media; flash memory devices; 
electrical, optical, acoustical or other forms of propagated signals (e.g., carrier 
waves, infrared signals, digital signals, etc.) and others. 
[001 2) An embodiment of a network device 8 to route packets of a network 

communication system is shown in Fig. 1. The network device 8 may comprise a 
network interface 10, a framer 11. one or more network processors 12/13, a 
switch fabric 1 4, and one or more external memories 1 5/1 6. Examples for the 
network device 8 may comprise an ATM switch (Asynchronous Transfer Mode), 
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an IP router (Internet Protocol), a SDH DXC (Synchronous Digital Hierarchy Data- 
cross Connection), and the like. 
[0013] The framer 1 1 may perform operations on frames. In an embodiment, the 
framer 1 1 may receive a line datagram from a network interface 10 of the network 
5 communication system, delimitate frames and extract payload, such as Ethernet 
packet from the frames. In another embodiment, the framer 1 1 may receive 
packets from network processor 13, encapsulate the packets into frames and map 
the frames onto the network interface 10. The framer 1 1 may further perform 
operations such as error detection and/or correction. Examples for the framer 1 1 
1 0 may comprise a POS (packet over Synchronous Optic Network) framer, a High- 

Level Data Link (HDLC) framer or the like. 
[0014] The network processors 12 and 13 may perform operations on packets. In 
an embodiment, the network processor 12 may process and forward the packets 
from the framer 1 1 to an appropriate port of another network device through the 
1 5 switch fabric 1 4. For example, the network processor 1 2 may assemble IPv4 
(Internet Protocol version 4) packets into CSIX (Common Switch Interface 
Specification) packets, modify packet headers and payloads, determine 
appropriate ports and forward the CSIX packets to the appropriate ports of the 
another network device. The network processor 13 may process and forward 
20 packets from the switch fabric 14 to appropriate ports 20 of the network interface 
1 0 through the framer 1 1 . For example, the network processor 1 3 may 
reassemble CSIX packets into IPv4 packets, modify packet headers and 
payloads. determine appropriate ports 20 and forward the IPv4 datagrams to the 
appropriate ports 20. Examples for the network processors 12 and 13 may 
25 comprise Intel® IXP 2XXX (e.g., IXP2400, IXP2800) network processors. 
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[001 5J The switch fabric 14 may receive and send packets from/to a network 

processor connected therewith. Examples for the switch fabric 14*nay comprise a 
switch fabric conforming to CSIX or other fabric technologies such as 
HyperTransport, Infiniband, PCI-X, Packet-Over-Synchronous Optical Network, 

5 RapldIO, and Utopia. 
[001 6] The external memories 15 and 1 6 may store entries 1 55/1 65 used by the 
network processors 12 and 13 to process and forward the packets. The entries 
may comprise node configuration data, queue configuration data, flow 
configuration data, network routing data, etc. The external memories 15 and 16 
10 may further buffer the packets. In one embodiment, the external memory 15/16 
may comprise SDRAM (Synchronous Dynamic Random Access memory) to store 
packets and QDR SRAM (Quad Data Rate Static Random Access Memory) to 
store entries. 

[001 7] Other embodiments may implement other modifications and variations on 

15 the structure of the network device as depicted in Fig. 1 . For instance, the network 
processors 12 and 13 may perform framing duties instead of the framer 1 1 and 
the switch fabric may be omitted in a single-box scenario. For another instance, 
the network processors 12 and 13 may be integrated as one. 

[0018] An embodiment of the network processor 1 2 (or network processor 1 3) is 
20 shown in Fig. 2. As shown, the network processor 1 2 may comprise a control 
plane 21 1 , a data plane 212 and a scratch pad 213 that are communicable with 
each other through a bus connection. 

[0019] The control plane 21 1 may be implemented as an integrated circuit (IC) 

with one or more processing cores 214 t ... 214 M such Intel® XScale® processing 

25 cores or StrongARM® processing cores to execute instructions to perform various 
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tasks. In an embodiment, the processing cores 214, ... 214 M of the control plane 
211 may execute Instructions to setup, configure and update entries 155/165 
stored in the external memories 15/16. For instance, the processing cores 214i ... 
21 4 M may update the external memories 15/16 which contain entries such as, for 
5 example, configuration data for nodes, configuration data for each packet queue, 
configuration data for each packet flow, etc. In another embodiment, the 
processing cores 21 4i . . . 21 4 M may further handle packets containing protocol 
message and routing information that may need relatively complex computations. 
For instance, the processing cores 214, .. . 214 M may process routing protocol 
1 0 packets containing routing Information such as, for example, RIP (Routing 

Information Protocol) packets, OSPF (Open Shortest Path First) packets, and the 
like. 

0020] The data plane 21 2 may comprise a plurality of microengines 21 5, . . . 21 5 N 
in Fig. 2 that may be communicable with each other. Each of the microengines 

1 5 may comprise a plurality of threads 21 61 ... 21 6k to process and forward packets 
and one or more local memories 21 8, ... 218 N to store instruction code 220 and 
entries 224. The local memory 218, ... 218 N may comprise a control store, a 
memory, general purpose registers, transfer registers, and/or other storage 
mechanisms. In an embodiment, the local memories 218, ... 218 N may comprise 

20 instruction code 220 executable by the threads 21 6, ... 21 6 K and one or more 
entries 224 consistent with the entries 155/165 of the external memories 15/16. 
The threads 21 61 . . . 21 6« may access the local memories 21 81 . . . 21 8 N to fetch 
some useful information for packet forwarding. Entries 155/165 may be cached 
from the external memory 15/16 to the local memories 218i ... 218 N of the 

25 microengines 21 5i . . . 21 5n based upon some criteria, for example, whether the 
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entries 155/165 are frequently used by one or more microengines 21 5i ... 21 5n of 
the data plane 212. Further, the entries 155/165 cached by one mlcroengine 21 5i 
... 215 N may be different from the entries 155/165 cached by another microengine 
215i...215 N . 

[0Q21] The scratch pad 213 is accessible by both the processing cores 214i ... 
21 4 M of the control plane 21 1 and the microengines 21 5, ... 21 5 N of the data 
plane 212. The scratch pad 213 may comprise a buffer 226i ... 226 N to store data 
for each microengines 215, ... 21 5 N . The buffers 226, ... 226 N may be 
implemented using various structures such as, for example, ring buffers, link lists, 
1 0 stacks, etc. In other embodiments, the scratch pad 21 5 may be regarded as a flat 
memory. 

10022] In an embodiment, processing cores 214i ... 214m of the control plane 21 1 
may update one or more entries 155/165 in the external memories 15/16 by 
adding, deleting or changing one or more entries 155/165, and may write 

1 5 'information related to the updated entries 1 55/1 65 to each buffer 226, . . . 226 N of 
the scratch pad 213 associated with a mlcroengine 215, ... 251 N that stores the 
updated entries 155/165 in its local memory 218, ... 21 8 N . Then, the microengines 
215, ... 21 5 N may extract information from its buffer 226i ... 226 N , read the 
updated entries 1 55/1 65 from the external memories 15/16 and update the 

20 corresponding entries 224 in the local memories 21 81 . . . 21 8n- The information 
written in the buffers 226, ... 226 N may comprise entry identifiers (e.g. addresses, 
entry numbers, entry pointers) that uniquely identify entries 155/165 of the 
external memories 15/16. 

[0023] Other embodiments may implement other modifications and variations on 

25 the structure of the network processor as depicted in Fig. 2. For example, the 
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network processor 1 2 may further comprise a hash engine, a peripheral 
component interconnect (PCI) bus interface for communicating, etc. 
[0024] Fig. 3 shows a process implemented by one or more processing cores 21 4i 
... 214 M of the control plane 211 to update an external entry 155/165 stored in an 
5 external memory 1 5/1 6. As shown, in block 301 , the control plane 21 1 may update 
an entry 155 in the external memory 15. Then, in block 302, the control plane 211 
may search for microengine(s) 215 t ... 215 N of the data plane 212 affected by the 
updated entry 155. In one embodiment, the control plane 211 determines a 
microenglne 215, ... 215 N is affected by the updated entry 155 by determining 

10 that the microengine 215i ... 215 N has the updated entry 155 cached in its 

corresponding local memory 21 81 . ... 21 8n. 
[0025] The control plane 21 1 may implement block 302 in various ways. In an 

embodiment, the control plane 21 1 may determine the affected microengines 21 5i 
... 215 N by referring to a table of the external memory 15/16 or scratch pad 213 

1 5 that lists the microengines 21 5i . . . 21 5 N having cached a particular external entry 
155. For example, the control plane 21 1 may supply a CAM (content addressable 
memory) .of the external memory 15 with an identifier (e.g. an address, index, 
hash value, etc.) for the updated entry 155 to obtain a list of microengines 21 5i ... 
21 5 N that have the entry 155 cached. In particular, the CAM may return a data 

20 word having at least N bits wherein each bit Indicates whether a corresponding 
microengine 215i ... 21 5 N has the updated entry 155 cached. However, it should 
be appreciated that the control plane 211 may utilize other techniques and 
structures to maintain a corresponds between entries 155/165 and the 
microengines 215^.. 21 5 N that have stored local copies of the entries 155/165. 
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[0026] In block 303, the control plane 21 1 may write information associated with 
the external entry 155 updated in block 301 to buffers 226i ... 226 N of 
microengines 215t ... 21 5 N affected by the updated entry 155. The information 
may comprise identifiers that identify external entry 155/165 that have been 
5 updated by the control plane 21 1 . 

[0027] For instance, if an entry 155 of external table 151 is updated in block 301 , 
the control plane 21 1 may search for the microengines 21 5i . . . 21 5n. that store 
the entry 1 55 in their local memories 21 8i . . . 21 8 N (block 302). Then, the control 
plane 21 1 in block 303 may write an identifier (e.g. address, entry number, entry 
1 0 pointer, and/or other data) for the updated entry 1 55 to the buffers 226i . . . 226 N of 
the affected microengines 21 5i ... 215 N identified in block 302. For example, if the 
control plane 21 1 determines in block 302 that microengines 21 5i and 21 5 N have 
cached the updated entry 155. then the control plane 21 1 in block 303 may write 
an identifier for the entry 1 55 to the corresponding buffers 226i and 226 N to inform 
15 the microengines 21 5i and 21 5 N that the identified entry 155 has been updated. 

I0028] In another embodiment, if all entries or more than a threshold level of 

entries of the external memory 15 are updated in block 301, the control plane 211 
may forgo block 302 and write a wildcard Identifier to the buffers 226 t ... 226 N 
indicating all cached entries 155 of the external memory 15 are invalid or 
20 outdated. 

[0029] Fig. 4 shows an embodiment of a method to update one ore more entries 
224 of local memories 218i ... 21 8 N of the data plane microengines 215i ... 215n. 
In block 402, one thread 216i ... 21 6« of each mlcroenglne 215i ... 215 N of the 
data plane 212 may be designated or otherwise configured to perform the task of 
25 updating cached entries 224 of the microenglne 21 5i ... 21 5n. In one 
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embodiment, the control plane 21 1 may designate a thread 216,... 216 K of each - 
microengine 21 5i ... 21 5 N that Is to update the cached entries 224 of the 
microengine 215, ... 215 N . Other embodiments may utilize other techniques to 
designate the thread to update the cached entries 224. For example, the 
5 microengine 21 5i ... 21 5 N may designate the thread, the thread may be 

predetermined by the instruction code 220, and/or the thread designation may be 
hardwired into the microengine 215, ... 215 N . In block 404, a thread 216, ... 216 K 
of a microengine 21 5, ... 21 5 N may be selected to continue executing its assigned 
tasks. Tothis end, the microengine 215, ... 215 N and/or the control plane 211 
1 0 may awaken and/or otherwise activate the selected thread using various thread 
scheduling algorithms such as, for example, round robin, priority, weighted 
priority, and/or other scheduling algorithms. 
0030] In block 406, the selected thread 216, ... 21 6 K may determine whether the 

selected thread 216, ... 216 K is designated to update the local memory 218, ... 
15 218 N of its microengine 215, ... 215 N . If selected thread 216, ... 216 K determines 
in block 406 that another thread 216, ... 216k is designated for updates, then the 
selected thread 216, ... 216 K in block 408 may continue to process packets in a 
normal fashion. If, however, the selected thread 216, ... 21 6 K Is designated to 
update its local memory 218, ... 218 N , then the thread 216, ... 216 K in block 410 
20 may determine whether the buffer 226, . . . 226 K for its microengine 21 5, ... 21 5 N 

indicates that entries 226 are invalid or outdated. 
[0031) The selected thread 216, ... 216« may implement block 410 in various 

ways. For an embodiment wherein the buffers 226, ... 226 N are scratch rings, the 
selected thread 216, ... 21 6 K may execute a predetermined instruction (e.g. 
25 'br_!inp_state[. . .]') of the instruction code 220 and may determine whether a 

10 
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returned value of the predetermined instruction is true (T) or false ('0'). The 
selected thread 21 61 ... 21 6 K may determine no updates are pending if the 
returned value is false, and likewise may determine one or more entries 226 of its 
local memory 218^.. 21 8 N are to be updated if the returned value Is true, 

[0632] If the selected thread 21 61 , . v 21 6 K determines to update entries 224 of its 
local memory 21 81 . . . 21 8 N , the thread 21 6i . . . 21 6 K in block 41 2 may extract 
identifiers for the updated entries 1 55/1 65 from the buffer 226i . . . 226 N associated 
with the microengine 215i ... 215 N of the thread 216i ... 216 K . The information 
may comprise an entry identifier that uniquely Identifies the updated entries 

1 0 1 55/1 65 of the external memories 1 5/1 6. Such an identifier may comprise an 
external memory number, an external memory pointer, an entry number, an entry 
pointer, and/or other identifying information from which an entry 155/165 may be 
discerned. However, if the selected thread 216! ... 21 6 K determines to update no 
entries 224 of its local memory 21 81 ... 21 8 N , the selected thread 21 61 ... 21 6 K - 

1 5 may continue to block 408 to perform normal packet processing. 

[0033] In block 414, the selected thread 216i ... 216 K may read entries 155/165 
from the external memory 15/16 that have been identified by information in its 
corresponding buffer 226i ... 226 N as being updated. Further, the selected thread 
21 61 ... 216k may update corresponding cached entries 224 based upon the 

20 entries read from the external memory 15/16 (block 416). 

[0034] Other embodiments may implement other modifications and variations to 
the process as depicted in Fig. 4. For example, a microengine 21 5i ... 21 5n may 
not assign a single thread 21 61 ... 21 6« to perform the task of updating local 
memory 218! ... 218 N . Instead, each thread 216! ... 216 K of the microengine 215i 
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... 215 N may determine whether to update entries 224 cached in its local memory 
21 81 ... 21 8n before continuing with normal packet processing. . 

[0035] A data flow diagram illustrating an embodiment of updating entries 224 of 
local memories 21 8 t ... 21 8 N of the network processor 12 is shown in Fig. 5. As 
5 shown, the control plane 21 1 may update one or more external entries 155 In an 
external memory 15 (arrow 501). Then, the control plane 211 may write 
information associated with the updated external entries 155 to the buffers 226i ... 
226 N assigned to the affected microengines 21 5i ... 21 5n (arrow 502). In 
response to a thread 21 61 ... 21 6« determining, based upon information stored in 

1 0 its buffers 21 81 ... 21 8 N , that one or more cached entries 224 of Its microengine 
215! ... 21 5 N have been updated, the thread 21 61 ...21 6k may read the updated 
external entries 1 55 from the external memory 1 5 (arrow 504) and update the 
corresponding local memory 21 81 ... 21 8n with the read entries 155 (arrow 505). 

[0036] While certain features of the invention have been described with reference 
15 to example embodiments, the description is not intended to be construed in a 
limiting sense. Various modifications of the example embodiments, as well as 
other embodiments of the invention, which are apparent to persons skilled in the 
art to which the invention pertains are deemed to lie within the spirit and scope of 
the Invention. 
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